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MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 

(FINANCIAL  MANAGEMENT  AND  COMPTROLLER) 

SUBJECT:  Report  on  Source  Selection  for  the  National  Polar-Orbiting  Operational 
Environmental  Satellite  System  -  Conical  Microwave  Imager/ Sounder 
(Report  No.  D-2006-097) 


We  are  providing  this  report  for  review  and  comment.  We  performed  this  audit  at 
the  request  of  the  Office  of  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics.  We  considered  management  comments  on  a  draft  of  this  report  in  preparing 
the  final  report. 

DoD  Directive  7650.3  requires  that  all  issues  be  resolved  promptly.  The  Air 
Force  comments  were  partially  responsive.  We  request  that  the  Military  Deputy,  Office 
of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition)  provide  additional  comments  on 
Recommendations  2  and  3  by  August  10,  2006. 

If  possible,  please  provide  management  comments  in  electronic  format  (Adobe 
Acrobat  file  only)  to  AudDFS@4odig.mil.  Copies  of  the  management  comments  must 
contain  the  actual  signature  of  the  authorizing  official.  We  cannot  accept  the  /  Signed  / 
symbol  in  place  of  the  actual  signature.  If  you  arrange  to  send  classified  comments 
electronically,  they  must  be  sent  over  the  SECRET  Internet  Protocol  Router  Network 
(SIPRNET). 

We  appreciate  the  courtesies  extended  to  the  staff.  Questions  should  be  directed 
to  Ms.  Lorin  T.  Pfeil  at  (703)  325-5568  (DSN  221-5568)  or  Mr.  Walter  Gaich  at  (703) 
325-5894  (DSN  221-5894).  See  Appendix  C  for  the  report  distribution.  The  team 
members  are  listed  inside  the  back  cover. 

By  direction  of  the  Deputy  Inspector  General  for  Auditing: 

P aul  J/Granetto ,  CPA 
Assistant  Inspector  General 
Defense  Financial  Auditing 
Service 
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Environmental  Satellite  System  -  Conical  Microwave 

Imager/Sounder 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  Acquisition  and  contracting  personnel 
within  DoD  and  the  military  departments  should  read  this  report.  It  discusses  the  need 
for  oversight  of  the  source  selection  process  for  contract  awards. 

Background.  This  audit  is  one  in  a  series  performed  at  the  request  of  the  Office  of 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics.  On  February  11, 
2005,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
referred  eight  Air  Force  contracts  for  DoD  Office  of  Inspector  General  (OIG)  to  audit  due 
to  concerns  that  the  Principal  Deputy  Assistant  Secretary  of  the  Air  Force,  Acquisition 
and  Management  may  have  used  undue  influence  to  award  these  contracts  to  Boeing 
Corporation  and  four  other  contractors.  One  contract  referred  for  audit  awarded  the 
National  Polar-Orbiting  Operational  Environmental  Satellite  System  -  Conical 
Microwave  Imager/Sounder.  That  contract  is  addressed  in  this  report.  Subsequently,  the 
DoD  OIG  initiated  additional  audits  based  on  the  same  concerns. 

Results.  The  Air  Force  did  not  award  the  National  Polar-Orbiting  Operational 
Environmental  Satellite  System  -  Conical  Microwave  Imager/Sounder  contract  in 
accordance  with  the  Federal  Acquisition  Regulation.  The  Air  Force  Source  Selection 
Authority  for  the  contract  award: 

•  lacked  impartiality  with  respect  to  the  contract  winner,  Boeing  Satellite  Systems 
and 

•  manipulated  complex  proposal  evaluation  ratings  to  benefit  Boeing  Satellite 
System’s  contract  proposal  and  hinder  Ball  Aerospace  and  Technologies 
Corporation’s  contract  proposal. 

In  addition,  source  selection  personnel  did  not  adequately  document  the  evaluation  of  the 
offerors’  proposals  and  used  undefined  and  inconsistently  applied  evaluation  ratings  in 
the  source  selection  reporting  process.  Also,  source  selection  personnel  chose 
significantly  more  relevant  contracts  from  Ball  Aerospace  and  Technologies  Corporation 
than  relevant  contracts  from  Boeing  Satellite  Systems  for  the  past  performance  evaluation 
of  the  source  selection  process.  As  a  result,  the  Air  Force  unfairly  awarded  the 
$317  million  Conical  Microwave  Imager/Sounder  contract  to  Boeing  Aerospace.  We 
recommend  that  the  Air  Force  establish  a  source  selection  oversight  process,  require 
support  for  source  selection  findings,  and  update  past  perfonnance  evaluation  guidance. 
See  the  Finding  section  of  the  report  for  detailed  recommendations. 


Management  Comments  and  Audit  Response.  The  Military  Deputy,  Office  of  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition)  concurred  with  two  of  the  three 
recommendations.  Comments  received  were  partially  responsive,  but  did  not  address  the 
need  for  a  sufficient  audit  trail  to  support  source  selection  actions.  The  Military  Deputy 
nonconcurred  with  the  recommendation  to  update  past  performance  evaluation  guidance 
to  include  methodology  to  address  when  there  are  significant  disparities  in  past 
performance  experience  between  offerors.  We  agree  that  past  performance  evaluation 
should  be  based  on  contractor  past  performance  as  a  whole.  However,  the  past 
performance  evaluation  methodology  needs  to  include  the  impact  of  past  performance 
strengths  and  weaknesses  in  proper  proportion  to  the  population  of  past  performance 
experiences  reviewed.  Therefore,  we  request  that  the  Secretary  of  the  Air  Force, 
Acquisition  and  Management  comment  on  this  report  by  August  10,  2006.  See  the 
Finding  section  of  the  report  for  a  discussion  of  management  comments  and  the 
Management  Comments  section  of  the  report  for  the  complete  text  of  the  comments. 
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Background 


In  accordance  with  Presidential  Decision  Directive  National  Science  and 
Technology  Council  (NSTC)-2,  May  5,  1994,  the  National  Polar-Orbiting 
Operational  Satellite  System  (NPOESS)  was  established  to  converge  DoD, 
Department  of  Commerce  (DoC),  and  National  Aeronautics  and  Space 
Administration  (NASA)  polar-orbiting  environmental  satellite  programs  into  a 
single  operational  system.  NPOESS  would  reduce  acquisition  and  operational 
costs  and  duplication  of  efforts  in  meeting  common  requirements  while  satisfying 
the  unique  requirements  of  the  civil  and  national  security  communities.  To 
implement  NSTC-2,  the  Integrated  Program  Office  (IPO),  consisting  of  personnel 
from  DoD,  DoC,  and  NASA,  was  created  to  manage,  plan,  develop,  fabricate,  and 
operate  NPOESS.  DoD  was  responsible  for  supporting  the  IPO  in  major  system 
acquisitions  necessary  to  support  NPOESS. 

Conical  Microwave  Imager/Sounder.  The  Conical  Microwave  Imager/Sounder 
(CMIS)  sensor  is  one  of  an  array  of  sensors  with  which  the  NPOESS  will  be 
equipped.  The  sensors  will  collect  and  disseminate  data  on  the  Earth’s  weather, 
atmosphere,  oceans,  land,  and  near  space  environment.  The  sensors  also  will  be 
able  to  monitor  the  entire  planet  and  provide  data  for  long-range  weather  and 
climate  forecasts  in  a  more  timely  and  accurate  manner.  The  CMIS  sensor  will 
collect  global  microwave  radiometry  and  sounding  data  to  produce  microwave 
imagery  and  other  meteorological  and  oceanographic  data  for  military  and 
civilian  use. 


CMIS 


ATMS 


VIIRS 


NPOESS  1330 
Configuration 


OMPS 


Figure  1.  NPOESS  with  CMIS  and  Other  Sensors 
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CMIS  Awards  and  Protest.  The  IPO  implemented  the  CMIS  acquisition 
strategy  in  two  phases.  Phase  I,  “Sensor  Definition  and  Risk  Reduction,” 
addressed  CMIS  sensor  payload  and  algorithm  development  up  to  preliminary 
design  review.  On  July  30,  1997,  the  Air  Force  awarded  Phase  I  contracts  to  both 
Boeing  Satellite  Systems  (BSS)  and  Ball  Aerospace  and  Technologies 
Corporation  (BATC).  Phase  II,  “Detailed  Design  and  Fabrication,”  addressed 
continued  sensor  development  up  to  critical  design  review  and  delivery  of  six 
CMIS  sensor  units.  On  July  30,  2001,  the  Air  Force  awarded  the  Phase  II 
contract  to  BSS  over  BATC.  On  August  9,  2001,  BATC  contested  the  CMIS 
Phase  II  contract  award  and  filed  a  complaint  with  the  Government 
Accountability  Office  (GAO)  Procurement  Law  Control  Group  stating  that  the 
Government  significantly  violated  applicable  regulations,  thereby  severely 
prejudicing  BATC.  On  September  21,  2001,  BATC  abruptly  withdrew  its  GAO 
protest  of  the  CMIS  award,  citing  business  reasons.  See  the  Appendix  B  for  a 
detailed  description  of  the  CMIS  Phase  II  source  selection  process. 

Darlene  Druyun  Case.  On  April  20,  2004,  Darlene  Druyun,  Principal  Deputy 
Assistant  Secretary  of  the  Air  Force,  Acquisition  and  Management  entered  a 
guilty  plea  in  Federal  court  for  conspiring  with  Boeing  Corporation’s1  (Boeing) 
chief  financial  officer  to  help  Boeing  win  a  multi-billion  dollar  airplane  tanker 
leasing  contract.  Druyun  admitted  in  a  post-plea  agreement  with  the  Government 
that  she  gave  Boeing  preferential  treatment  on  numerous  other  contracts  as  well. 
On  February  11,  2005,  the  Acting  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  referred  eight  Air  Force  contract  awards  made  by 
Druyun,  including  the  CMIS  Phase  II  contract  award  for  which  Druyun  was  the 
Source  Selection  Authority  (SSA),  to  the  DoD  OIG  for  audit. 


Objective 


Our  audit  objective  was  to  determine  whether  the  National  Polar-Orbiting 
Operational  Environmental  Satellite  System  -  Conical  Microwave  Imager  Sensor 
(NPOESS-CMIS)  was  procured  in  accordance  with  the  Federal  Acquisition 
Regulation  (FAR).  The  scope  of  our  audit  was  limited  to  source  selection  for  the 
CMIS  Phase  II  contract.  See  Appendix  A  for  a  discussion  of  the  scope  and 
methodology  and  prior  coverage  related  to  the  objective. 


Managers’  Internal  Control  Program 


DoD  Directive  5010.38,  “Management  Control  Program,”  August  26,  1996,  and 
DoD  Instruction  5010.40,  “Management  Control  Program  Procedures,” 

August  28,  1996,  require  DoD  organizations  to  implement  a  comprehensive 
system  of  management  controls  that  provides  reasonable  assurance  that  programs 
are  operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls.  We  did 
not  review  the  Air  Force  management  control  program  because  the  audit  focused 
on  detennining  whether  the  CMIS  contract  was  awarded  in  accordance  with  the 


1  Parent  corporation  of  Boeing  Satellite  Systems. 
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FAR.  We  did,  however,  become  aware  of  internal  control  weaknesses  over  the 
source  selection  process.  Recommendations  1.,  2.,  and  3.  should  help  to 
strengthen  those  controls. 
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CMIS  Phase  II  Contract  Award 


The  Air  Force  did  not  award  the  CMIS  Phase  II  contract  in  accordance 
with  the  FAR.  Specifically,  the  SSA  for  the  CMIS  Phase  II  contract 
award: 


•  lacked  impartiality  with  respect  to  the  contract  award  winner,  BSS, 
and 

•  manipulated  complex  proposal  evaluation  ratings  to  benefit  BSS’ 
contract  proposal  and  hinder  BATC’s  contract  proposal. 

In  addition,  source  selection  personnel  did  not  adequately  document  the 
evaluation  of  the  offerors’  proposals  and  used  undefined  and 
inconsistently  applied  evaluation  ratings  in  the  source  selection  reporting 
process.  Also,  source  selection  personnel  chose  significantly  more 
relevant  contracts  from  BATC  than  relevant  contracts  from  BSS  for  the 
past  performance  evaluation  of  the  source  selection  process. 

These  conditions  occurred  because  the  Air  Force  did  not  ensure  the 
implementation  of  appropriate  FAR  and  Air  Force  policies  and  procedures 
for:  the  impartial,  fair,  and  equitable  treatment  of  the  contract  offerors; 
sufficient  documentation  to  support  actions  taken;  and  precise  proposal 
evaluation  reporting.  Also,  Air  Force  policy  on  past  performance 
evaluation  methodology  does  not  address  when  there  are  significant 
disparities  in  past  performance  experience  between  offerors.  As  a  result, 
the  Air  Force  unfairly  awarded  the  $317  million  CMIS  Phase  II  contract  to 
BSS. 


CMIS  Phase  II  Source  Selection  Plan 


In  accordance  with  FAR  Part  15  and  Air  Force  Federal  Acquisition  Regulation 
Supplement  (AFFARS)  Part  5315,  the  Air  Force  prepared  and  executed 
requirements  set  forth  in  the  Source  Selection  Plan  (SSP),  “National  Polar- 
Orbiting  Operational  Environmental  Satellite  System  -  Sensor  Payload  & 
Algorithm  Development  Down-Selection  for  the  Conical  Microwave 
Imager/Sounder,”  January  2,  2001.  The  SSP  listed  the  CMIS  Phase  II  source 
selection  organization  and  explained  how  each  offeror’s  proposal  was  to  be 
evaluated.  See  Appendix  B  for  detailed  information  on  the  CMIS  Phase  II  source 
selection  plan. 


CMIS  Source  Selection  Ratings 


The  Air  Force  did  not  award  the  NPOESS-CMIS  contract  in  accordance  with  the 
FAR.  The  CMIS  Phase  II  SSA  lacked  impartiality  with  respect  to  the  contract 
award  winner,  BSS,  and  manipulated  the  complex  proposal  evaluation  ratings  to 
benefit  the  BSS  contract  proposal  and  hinder  the  BATC  contract  proposal.  In 
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addition,  source  selection  personnel  did  not  adequately  document  the  evaluation 
of  the  offerors’  proposals  and  used  undefined  and  inconsistently  applied 
evaluation  ratings  in  the  source  selection  reporting  process.  Also,  source 
selection  personnel  chose  significantly  more  relevant  contracts  from  BATC  than 
relevant  contracts  from  BSS  for  the  past  performance  evaluation  of  the  source 
selection  process. 

SSA  Impartiality.  FAR  Part  3,  “Improper  Business  Practices  and  Personal 
Conflicts  of  Interest,”  states  that  “government  business  shall  be  conducted  in  a 
manner  above  reproach  and  with  complete  impartiality  and  with  preferential 
treatment  for  none.  The  general  rule  is  to  avoid  strictly  any  conflict  of  interest  or 
even  the  appearance  of  a  conflict  of  interest.”  However,  the  SSA  lacked 
impartiality  with  respect  to  BSS  in  the  CMIS  Phase  II  source  selection.  In  late 
2000,  the  SSA’s  daughter  and  future  son-in-law  were  hired  to  work  at  Boeing.  In 
the  United  States  District  Court  for  the  Eastern  District  of  Virginia, 

“Supplemental  Statement  of  Facts,”  for  criminal  case  #04-150-A,  the  SSA 
acknowledged  that  Boeing’s  employment  of  her  daughter  and  future  son-in-law 
was  at  the  SSA’s  request,  and  their  employment  influenced  her  decisions  in 
matters  affecting  Boeing.  The  SSA’s  daughter  and  future  son-in-law  were  still 
employed  by  Boeing  during  the  CMIS  Phase  II  source  selection  process.  FAR 
Part  3  also  states  that  the  official  conduct  of  Government  personnel  must  be  such 
that  they  would  have  no  reluctance  to  make  a  full  public  disclosure  of  their 
actions.  However,  the  contracting  officer  did  not  provide  a  required  signed 
Conflict  of  Interest  statement  and  Nondisclosure  Agreement  for  the  SSA.  In  this 
environment,  the  SSA  could  not  be  relied  on  to  make  an  objective  assessment  of 
the  CMIS  Phase  II  source  selection. 

Changes  to  Final  Evaluation  Ratings.  The  SSA  manipulated  the  final  proposal 
evaluation  ratings  to  benefit  the  BSS  contract  proposal  and  hinder  the  BATC 
contract  proposal.  In  late  June  2001,  the  Source  Selection  Evaluation  Team 
(SSET)  completed  its  final  evaluation  of  the  BATC  and  BSS  proposals  and 
prepared  the  Proposal  Analysis  Report  with  the  final  evaluation  matrix  (Table  1). 
The  final  evaluation  matrix  for  both  offerors  showed  that  BATC  was  rated  better 
in  the  significantly  more  important  Mission  Capability  factor,  while  BSS  was 
rated  better  in  the  less  important  Proposal  Risk  and  Cost  factors.  Both  BATC  and 
BSS  were  given  “Significant  Confidence”  ratings  in  the  significantly  more 
important  Past  Performance  factor. 
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Table  1.  Final  SSET  Evaluation  Matrix 

(data  obtained  from  Proposal  Analysis  Report) 

BATC  (Ball) 

BSS  (Boeing) 

Subfactors 

Factors 

System 

Optimization 

Sensor 

Design 

Algorithm 

SEIT 

Program 

Execution 

System 

Optimization 

Sensor 

Design 

Algorithm 

SEIT 

Program 

Execution 

Past  Performance 

High  Confidence,  Significant  Confidence, 

Confidence,  Little  Confidence,  No  Confidence 

Significant  Confidence 

Significant  Confidence 

Mission  Capability 

Blue  (Exceptional),  Green  (Acceptable), 

Yellow  (Marginal),  Red  (Unacceptable) 

Blue* 

Blue 

Green 

Blue 

Green 

Green* 

Green 

Green* 

Green 

Green 

Proposal  Risk 

High,  Moderate  High,  Moderate, 

Low  Moderate,  Low 

Moderate 

High 

Moderate 

High 

Moderate* 

Low 

Low 

Low 

Moderate 

Low* 

Moderate 

Low 

Cost 

Propc 

$308. 

>sed 

PC**  $319. 9M 

Proposed 
$309. 1M 

PC  $316. 6M 

4M 

Reasonable  Realism 

Reasonable  Realism 

*SSET  Ratings  chosen  and  tasked  by  SSA  for  SSAC  review 

**Probable  Cost 

On  June  29,  2001,  the  SSET  and  Source  Selection  Advisory  Council  (SSAC) 
briefed  the  SSA  on  the  CMIS  Phase  II  final  proposal  evaluation  ratings 
documented  in  the  Proposal  Analysis  Report.  At  the  briefing,  the  attendees 
concluded  that  the  BSS  and  BATC  proposals  were  very  close  and  that  there  was 
no  consensus  winner  for  the  CMIS  source  selection.  The  SSA  tasked  the  SSAC 
to  address  the  basis  of  five  specific  SSET  subfactor  ratings  (shaded  ratings  in 
Table  1).  On  July  9,  2001,  the  SSAC  responded  in  the  SSAC  Addendum  to  the 
Proposal  Analysis  Report  by  concurring  with  the  SSET  basis  on  four  of  the  five 
subfactor  ratings  and  non-concurring  with  one  rating.  The  SSA  reviewed  the 
SSAC  response  along  with  the  Proposal  Analysis  Report  and  SSAC  Addendum 
and  made  changes  to  the  proposal  ratings  for  both  BSS  and  BATC  (Table  2).  On 
July  30,  2001,  the  SSA  made  the  CMIS  Phase  II  source  selection  and  chose  BSS 
over  BATC. 
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Table  2.  SSA  Actions  on  Selected  Subfactor  Ratings 

(data  obtained  from  SSAC  Addendum  to  Proposal  Analysis  Report  and  Source  Selection 

Decision  Document) 

Offeror 

Factor/ 

Subfactor 

SSET 

Conclusion 

SSAC 

Conclusion 

SSA  Change 

1  - BATC 

Proposal  Risk/ 
Algorithm 

Moderate  Risk 

Concur 

Raised  to 
Moderate -High 

2  - BATC 

MC*/System 

Optimization 

Blue 

Concur 

Lowered  to 
Green 

3 -BSS 

MC/System 

Optimization 

Green 

Concur 

Raised  to  Blue 

4 -BSS 

MC/ 

Algorithm 

Green 

Raised  to  Blue 

Accepted 

Change 

5 -BSS 

Proposal  Risk/ 
Algorithm 

Low  Risk 

Concur 

Accepted 

^Mission  Capability 

From  our  analysis  of  the  SSA  briefing  notes,  Proposal  Analysis  Report,  SSAC 
Addendum  to  the  Proposal  Analysis  Report,  and  Source  Selection  Decision 
Document,  we  identified  the  following  examples  of  how  the  SSA  manipulated  the 
final  proposal  ratings  for  both  BSS  and  BATC. 

SSAC  Analysis  of  SSET  Ratings.  The  SSA  selectively  accepted  or 
rejected  SSAC  conclusions  of  specific  SSET  subfactor  ratings  to  improve  the  BSS 
proposal  or  detrimentally  affect  the  BATC  proposal.  The  CMIS  sensor  is  the 
most  complex  sensor  on  the  NPOESS  satellite.  The  CMIS  SSET  subfactor 
evaluation  teams  who  reviewed  the  BATC  and  BSS  proposals  were  headed  by 
non-Govemment  personnel  who  were  members  of  the  Federally  Funded  Research 
and  Development  Corporation.  The  members  of  the  System  Optimization  and 
Algorithm  subfactor  teams  were  doctoral-degreed  personnel  with  expertise  in  the 
space  science  field.  The  SSAC  concurred  with  the  SSET  basis  for  four  of  the  five 
subfactor  ratings  and  non-concurred  with  one  rating.  However,  in  making  the 
source  selection  decision,  the  SSA  disregarded  the  advice  of  both  the  SSET  and 
SSAC  and  changed  three  of  the  subfactor  ratings,  even  though  the  SSAC 
concurred  with  the  basis  of  the  SSET  ratings  for  these  subfactors  (Table  2, 

Rows  1-3).  In  contrast,  the  SSA  accepted  a  SSAC  finding  to  non-concur  with  the 
SSET  and  raise  the  BSS  Algorithm  subfactor  rating  (Table  2,  Row  4).  The 
common  theme  was  that  in  these  four  situations,  the  BSS  proposal  was  improved 
or  the  BATC  proposal  was  weakened.  On  the  fifth  item,  the  SSA  accepted  the 
SSET  and  SSAC  concurrence  on  the  BSS  Algorithm  Proposal  Risk  subfactor 
rating  of  low  risk  (Table  2,  Row  5).  However,  the  SSA  had  stated  in  the  June  29, 
2001  decision  briefing  that  the  BSS  Science  Algorithm  proposal  was  a  strength, 
calling  into  question  why  the  SSA  would  task  the  SSAC  to  address  this  area  in  the 
first  place. 

Other  SSET  Ratings.  The  SSA  did  not  address  the  basis  of  an  SSET 
subfactor  rating  for  BSS  with  significant  weaknesses  that  could  have  led  to  a 
detrimental  ratings  change  for  BSS.  The  SSET  stated  in  the  Proposal  Analysis 
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Report  that  BSS  had  two  significant  weaknesses  in  the  Systems  Engineering, 
Integration,  and  Testing  (SEIT)  subfactor-one  significant  weakness  was  in  an 
area  that  BSS  did  not  adequately  explain  and  the  other  significant  weakness  was 
in  an  area  that  BSS  incompletely  characterized  because  it  did  not  seem  to 
understand  the  particular  requirements  of  the  SEIT  subfactor.  The  SSET  also 
stated  that  BSS  had  a  third,  though  not  significant,  weakness  in  SEIT.  The  SSET 
gave  BSS  a  “Moderate”  Proposal  Risk  rating  for  SEIT.  Yet,  the  SSA  did  not  task 
the  SSAC  to  address  the  basis  for  the  S SET’s  “Moderate”  rating  instead  of 
possibly  a  “Moderate-High”  rating  in  light  of  the  significant  weaknesses.  In 
contrast,  the  SSA  tasked  the  SSAC  to  question  the  basis  of  the  S SET’s  rating  for 
BSS’  Science  Algorithms  subfactor  with  only  two  insignificant  weaknesses 
(Table  2,  Row  5).  At  the  SSA  decision  briefing,  the  SSA  stated  that  BSS’ 

Science  Algorithm  subfactor  was  a  strength. 

Past  Performance  Evaluation.  The  SSA  did  not  address  an  SSET 
concern  about  the  disparity  in  number  of  contracts  reviewed  in  the  CMIS  Phase  II 
Past  Performance  evaluation.  The  Proposal  Analysis  Report  included  a  statement 
that  BATC  had  significantly  more  “highly  relevant”  contract  experience,  “leading 
to  more  areas  for  the  Government  to  find  problems”  with  BATC’s  past 
performance.  The  SSA  did  not  address  this  concern  with  either  the  SSET  or 
SSAC.  In  making  the  source  selection  decision,  the  SSA  lowered  BATC’s  Past 
Performance  rating  from  “Significant  Confidence”  to  “Confidence”  primarily  on 
the  basis  of  what  the  SSET  called  a  slight  ratings  difference  between  BSS  and 
BATC  in  the  Science  Algorithm  area  of  the  Past  Perfonnance  evaluation. 

Source  Selection  Plan  Requirements.  The  SSA  misinterpreted 
requirements  in  the  SSP  that  benefited  the  BSS  proposal.  The  SSP  stated  that  in 
rating  the  offerors’  proposals,  the  Mission  Capability  Factor  was  to  be 
significantly  more  important  than  the  Proposal  Risk  Factor.  However,  in  making 
the  source  selection  decision,  the  SSA  concluded  that  BSS  had  a  better  overall 
sensor  design  than  BATC  because  the  advantage  of  BSS  over  BATC  (Moderate 
versus  Moderate-High  risk)  in  the  sensor  design  Proposal  Risk  Factor  offset  the 
advantage  of  BATC  over  BSS  (blue  versus  green)  in  sensor  design  Mission 
Capability  Factor.  This  logic  contradicts  proposal  evaluation  instructions  set 
forth  in  the  SSP. 

The  SSA  decisions  made  regarding  the  SSET’s  final  proposal  ratings  appear  to 
favor  BSS  over  BATC  and  call  into  question  the  SSA’s  impartiality  with  respect 
to  the  CMIS  Phase  II  source  selection  process.  Table  3  shows  an  evaluation 
matrix  after  the  SSA’s  changes. 
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Table  3.  Evaluation  Matrix  after  SSA  Changes 

(data  obtained  from  Source  Selection  Decision  Document) 

BATC  (Ball) 

BSS  (Boeing) 

Subfactors 

Factors 

System 

Optimization 

Sensor 

Design 

Algorithm 

SEIT 

Program 

Execution 

System 

Optimization 

Sensor 

Design 

Algorithm 

SEIT 

Program 

Execution 

Past  Performance 

High  Confidence,  Significant  Confidence, 

Confidence,  Little  Confidence,  No  Confidence 

Confidence 

Significant  Confidence 

Mission  Capability 

Blue  (Exceptional),  Green  (Acceptable), 

Yellow  (Marginal),  Red  (Unacceptable) 

Green 

Blue 

Green 

Blue 

Green 

Blue 

Green 

Blue 

Green 

Green 

Proposal  Risk 

High,  Moderate-High,  Moderate, 

Low  Moderate,  Low 

Moderate 

High 

Moderate 

High 

Moderate 

High 

Low 

Low 

Low 

Moderate 

Low 

Moderate 

Low 

Cost 

PropoS' 

$308.4 

ed 

PC  $319. 9M 

Proposed 

$309. 1M 

PC  $316. 6M 

M 

Reasonable  Realism 

Reasonable  Realism 

Supporting  Documentation.  Source  selection  personnel  did  not  adequately 
document  evaluation  results  in  the  source  selection  reporting  process.  FAR 
Part  4,  Subpart  4.801(b)  states  that,  “documentation  in  the  contract  files  shall  be 
sufficient  to  constitute  a  complete  history  of  the  transaction  for  the  purpose  of 
providing  a  complete  background  as  the  basis  for  informed  decisions  at  each  step 
in  the  acquisition  process  and  supporting  actions  taken.”  SSET  and  SSAC 
personnel  prepared  the  Proposal  Analysis  Report  to  document  the  results  of  the 
evaluations  and  comparative  analysis  of  offerors’  proposals.  Although  supporting 
documentation  was  available  in  the  CMIS  Phase  II  contract  files,  we  were  unable 
to  trace  the  final  proposal  evaluation  ratings  in  the  Proposal  Analysis  Report  back 
to  supporting  documentation  in  the  contract  files  because  of  the  lack  of  a  clear 
audit  trail.  Also,  the  SSA  documented  the  contract  award  selection  of  BSS  in  the 
Source  Selection  Decision  Document,  but  did  not  provide  specific  support  for 
why  she  changed  three  of  the  sub  factor  ratings-ratings  that  both  the  SSAC  and 
SSET  agreed  on. 

Source  Selection  Reporting.  Source  selection  personnel  also  used  undefined 
and  inconsistently  applied  evaluation  ratings  in  the  source  selection  reporting 
process.  For  example,  the  SSAC  used  an  undefined  color  code  rating  to  rate  a 
subfactor  in  the  SSAC  Addendum  to  the  Proposal  Analysis  Report.  As  stated  in 
the  SSP,  the  color  codings  for  the  subfactor  evaluation  ratings  for  the  Past 
Perfonnance  and  Mission  Capability  factors  were:  “Blue,”  “Green,”  “Yellow,” 
and  “Red.”  However,  the  SSAC  gave  BSS  a  “High  Green”  to  rate  a  subfactor  in 
the  SSAC  Addendum  to  the  Proposal  Analysis  Report.  In  reviewing  lessons 
learned  from  the  CMIS  source  selection,  the  Air  Force  acknowledged  that  the  use 
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of  shades  of  colors  could  cause  problems  in  interpreting  proposal  conclusions.  In 
another  example,  the  SSET  made  an  inconsistent  statement  in  summarizing  the 
BSS  proposal  in  the  Proposal  Analysis  Report.  The  SSET  gave  BSS  “Green”  or 
acceptable  ratings  for  each  of  the  System  Optimization  and  Algorithm  subfactors. 
However,  in  the  Proposal  Analysis  Report  summary,  the  SSET  stated  that  BSS 
was  rated  “Blue”  or  exceptional  for  the  same  subfactors.2 

Past  Performance.  Source  selection  personnel  chose  significantly  more  relevant 
contracts  from  BATC  than  relevant  contracts  from  BSS  for  the  past  performance 
factor  evaluation  of  the  source  selection  process.  The  SSET  Chairperson 
appointed  the  Perfonnance  Risk  Assessment  Group  (PRAG)  to  evaluate  relevant 
current  and  past  perfonnance  to  assess  confidence  in  the  ability  of  each  offeror  to 
meet  the  requirements  of  the  source  selection.  In  total,  the  PRAG  chose  33 
BATC  and  BATC  subcontractor  contracts,  compared  to  13  BSS  and  BSS 
subcontractor  contracts  for  the  past  perfonnance  evaluation  (see  Table  4). 


Table  4.  Quantity  of  Contracts  Chosen  for  Past  Performance  Evaluation 

(BATC  had  seven  subcontractors  and  BSS  had  five 
subcontractors  considered  for  the  CMIS  project) 

Offeror 

Prime 

Sub-Contractor 

Total 

BATC 

13 

20 

33 

BSS 

8 

5 

13 

The  Proposal  Analysis  Report  included  a  statement  that  the  significant  disparity 
in  past  contract  experience  led  “to  more  areas  for  the  Government  to  find 
problems”  with  BATC’s  past  perfonnance.  Auditors  were  unable  to  detennine 
the  intent  of  the  statement  because  of  a  lack  of  supporting  documentation.  In 
addition,  SSET  personnel  interviewed  could  not  recall  specifics  about  the 
statement.  While  auditors  saw  no  evidence  that  the  significant  disparity  in  past 
contract  experience  was  prejudicial  to  BATC’s  past  perfonnance  evaluation 
ratings,  such  disparities  could  be  prejudicial  to  contract  offerors  in  other  closely 
matched  source  selections.  In  making  the  CMIS  source  selection  decision,  the 
SSA  downgraded  BATC’s  entire  Past  Performance  evaluation  rating  from 
“Significant  Confidence”  to  “Confidence”  primarily  on  the  basis  of  what  the 
SSET  called  a  slight  ratings  difference  in  the  Science  Algorithm  area  (see  Table 

3).3 


2  These  two  BSS  subfactor  ratings  were  also  changed  by  the  SSA  from  “Green”  to  “Blue”  after  the  source 
selection  briefing. 

3  The  CMIS  Past  Performance  evaluation  covered  six  areas:  the  five  Mission  Capability  sub  factors 
(System  Optimization,  Sensor  Design,  Science  Algorithm,  SEIT,  and  Program  Execution)  and  Cost 
Performance. 
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Air  Force  Implementation  of  Policies  and  Procedures 


The  Air  Force  did  not  ensure  the  implementation  of  appropriate  FAR  and  Air 
Force  policies  and  procedures  for  the  impartial,  fair,  and  equitable  treatment  of 
contract  offerors;  sufficient  documentation  to  support  actions  taken;  and  precise 
proposal  evaluation  reporting.  In  addition,  Air  Force  policy  on  past  performance 
evaluation  for  source  selections  does  not  provide  a  methodology  to  address  when 
there  are  significant  disparities  in  past  performance  experience  between  offerors. 

Treatment  of  Contract  Offerors.  The  Air  Force  did  not  ensure  the 
implementation  of  policies  and  procedures  for  the  fair,  impartial,  and  equitable 
treatment  of  contract  offerors.  FAR  Part  1,  Subpart  602-2,  “Contracting  Officer 
Responsibilities,”  states  that  contracting  officers  are  responsible  for  ensuring 
performance  of  all  necessary  actions  for  effective  contracting  and  for 
safeguarding  the  interests  of  the  United  States  in  its  contractual  relationships.  In 
order  to  perform  these  responsibilities,  contracting  officers  should  be  allowed 
wide  latitude  to  exercise  business  judgment  and  shall: 

•  ensure  that  all  requirements  of  law,  executive  orders,  regulations,  and  all 
other  applicable  procedures,  including  clearances  and  approvals,  have 
been  met  before  a  contract  shall  be  entered  into; 

•  ensure  that  contractors  receive  impartial,  fair,  and  equitable  treatment;  and 

•  request  and  consider  the  advice  of  specialists  in  audit,  law,  and  other 
fields,  as  appropriate. 

However,  the  CMIS  contracting  officer  did  not  invoke  FAR  Part  1  to  investigate 
the  following  circumstances  that  could  have  provided  the  basis  to  call  into 
question  the  SSA’s  impartiality  with  respect  to  the  CMIS  Phase  II  source 
selection. 

•  The  SSA  did  not  complete  a  Conflict  of  Interest  Statement  or  Non- 
Disclosure  Agreement. 

•  The  SSA  changed  specific  evaluation  ratings  to  benefit  one  offeror  over 
the  other. 

•  The  position  of  the  SSA’s  immediate  superior,  Assistant  Secretary  of  the 
Air  Force  for  Acquisition  and  Management,  was  vacant  during  the  final 
2  months  of  the  source  selection  process,  and  thus  the  SSA  was  acting  in 
that  role. 

Training  contracting  officers  to  ensure  familiarity  with  their  responsibilities  and 
authority  will  help  them  be  alert  to  suspicious  circumstances  in  source  selection 
actions. 

Documentation  of  Supporting  Actions.  The  Air  Force  did  not  ensure  the 
implementation  of  policies  and  procedures  for  sufficient  documentation  to  support 
actions  taken  by  source  selection  personnel.  Although  there  appeared  to  be 
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supporting  documentation  available  in  the  CMIS  Phase  II  contract  files,  we  were 
unable  to  validate  the  final  proposal  evaluation  ratings  in  the  Proposal  Analysis 
Report  by  tracing  final  evaluation  data  back  to  supporting  documentation  because 
of  the  lack  of  a  clear  audit  trail.  The  lack  of  a  clear  audit  trail  calls  into  question 
the  validity  of  information  provided  in  the  Proposal  Analysis  Report  and  used  by 
the  SSA  to  document  the  source  selection  decision  for  the  CMIS  Phase  II  contract 
award.  Incorporating  guidance  into  Air  Force  policy  to  require  an  audit  trail  that 
will  support  source  selection  findings  will  help  to  ensure  a  complete  history  of  the 
source  selection  process. 

Evaluation  Reporting.  AFFARS  5315,  Part  53 15.304,  Air  Force  Informational 
Guidance  5315.305,  “Proposal  Evaluation,”  July  6,  2000,  states  that  the  Proposal 
Analysis  Report  narrative  assessment  of  the  offerors’  proposal  evaluations  must 
be  precise.  However,  the  CMIS  Phase  II  Proposal  Analysis  Report  and  SSAC 
Addendum  to  the  Proposal  Analysis  Report  included  undefined  and  inconsistently 
applied  color  ratings.  These  types  of  errors  could  lead  to  evaluation  interpretation 
problems  and  mislead  the  SSA  when  making  source  selection  decisions. 
Implementing  an  independent  quality  review  of  relevant  source  selection 
documents  will  help  to  minimize  evaluation  reporting  errors. 

Past  Performance  Methodology.  Air  Force  policy  on  past  performance 
evaluation  for  source  selections  does  not  provide  a  methodology  to  address 
situations  when  there  are  significant  disparities  in  past  perfonnance  experience 
between  offerors.  In  total,  the  PRAG  chose  33  BATC  and  BATC  subcontractor 
contracts,  compared  to  13  BSS  and  BSS  subcontractor  contracts  for  the  CMIS 
Phase  II  past  performance  evaluation.  The  Proposal  Analysis  Report  included  a 
statement  that  the  significant  disparity  led  “to  more  areas  for  the  Government  to 
find  problems”  with  BATC’s  past  performance.  We  did  not  identify  Air  Force 
procedures  that  take  into  account  situations  when  a  disparate  number  of  contracts 
are  chosen  for  past  performance  evaluations  of  offerors.  However,  in  a  source 
selection  as  close  as  the  CMIS  Phase  II  source  selection,  significant  disparities  in 
past  performance  experience  can  lead  the  SSA  to  prejudicially  rate  an  offeror  for 
past  performance.  Adding  methodology  to  Air  Force  policy  to  address  situations 
when  there  are  any  significant  disparities  between  offerors  will  help  to  ensure  a 
fair  and  balanced  past  performance  evaluation  in  the  source  selection  process. 


Summary 


The  Air  Force  did  not  award  the  CMIS  Phase  II  contract  in  accordance  with  the 
FAR.  The  Air  Force  did  not  ensure  implementation  of  appropriate  FAR  and  Air 
Force  policies  and  procedures  for:  the  impartial,  fair,  and  equitable  treatment  of 
contract  offerors;  sufficient  documentation  to  support  actions  taken;  and  precise 
proposal  evaluation  reporting.  In  addition,  Air  Force  policy  on  past  performance 
evaluation  for  source  selections  does  not  provide  a  methodology  to  address 
situations  when  there  are  significant  disparities  in  past  perfonnance  experience 
between  offerors.  As  a  result,  the  Air  Force  unfairly  awarded  the  $317  million 
CMIS  Phase  II  contract  to  Boeing  Satellite  Systems. 
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Recommendations,  Management  Comments,  and  Audit 
Response 


We  recommend  that  the  Assistant  Secretary  of  the  Air  Force  (Acquisition 
and  Management): 

1.  Establish  a  source  selection  oversight  process  that  will  ensure: 

a.  Contracting  officers’  familiarity  with  their  authority  and 
responsibilities  set  forth  in  FAR  Part  1,  Subpart  1.6. 

Management  Comments.  The  Military  Deputy,  Office  of  the  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  concurred  and  stated  that  in  response  to 
previous  audits  and  recommendations  of  the  Defense  Science  Board  Task  Force, 
the  Air  Force  has  updated  contract  clearance  procedures  to  ensure  additional 
oversight  and  independent  review  of  contract  actions.  This  oversight  will  always 
be  at  least  one  level  above  the  contracting  officer  to  ensure  that  the  procedures  in 
FAR  Part  1,  Subpart  1.6,  as  well  as  all  other  relevant  contracting  policies  are 
being  followed.  These  procedures  were  updated  in  Air  Force  Acquisition 
Circular  (AFAC)  2005-0810  on  August  10,  2005. 

Audit  Response.  Management  comments  are  responsive.  AFAC  2005-0810, 
effective  September  10,  2005,  updated  Air  Force  procedures. 

b.  Independent  qualitative  review  of  relevant  source  selection 
documents. 

Management  Comments.  The  Military  Deputy,  Office  of  the  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  concurred  and  stated  that  in  addition  to 
the  clearance  process  changes  addressed  in  response  to  Recommendation  l.a.,  the 
Air  Force  recently  updated  legal  review  responsibilities  to  ensure  the  participation 
of  supporting  legal  offices  to  provide  advice  and  coordination  for  source  selection 
decisions,  and  to  review  key  documents,  including  those  involving  source 
selection.  This  change  to  the  legal  review  procedures  was  issued  in  AFAC  2006- 
0329  on  March  29,  2006. 

Audit  Response.  Management  comments  are  responsive.  The  implementation  of 
the  revised  guidance  began  on  May  1,  2006. 

2.  Update  Air  Force  policy  to  require  an  audit  trail  to  support  source 
selection  findings. 

Management  Comments.  The  Military  Deputy,  Office  of  the  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  concurred  and  stated  that,  as  discussed  in 
Recommendation  l.a.,  AFAC  2005-0810  incorporated  several  changes  to  the 
source  selection  process.  The  changes  included  a  requirement  for  the  source 
selection  evaluation  team  (SSET)  to  provide  a  source  selection  recommendation 
to  the  source  selection  authority  (SSA).  Other  changes  included  the  retention  of 
all  relevant  records  supporting  the  SSET  recommendation  to  ensure  there  is  an 
audit  trail  of  source  selection  evaluations  after  the  information  is  presented  in  any 
form  to  the  SSA. 
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Audit  Response.  Management  comments  are  partially  responsive.  AFAC  2005- 
OS  10  added  the  retention  of  records  requirement  to  AFFARS  Mandatory 
Procedures  53 15.3.  However,  this  requirement  does  not  address  the  need  for  a 
sufficient  audit  trail.  We  request  that  the  Air  Force  add  a  statement  to  AFFARS 
Mandatory  Procedures  5315.3  requiring  that  an  audit  trail  exist  between  source 
selection  conclusions  and  their  supporting  records. 

3.  Update  past  performance  evaluation  guidance  to  include  methodology  to 
address  when  there  are  significant  disparities  in  past  performance 
experience  between  offerors. 

Management  Comments.  The  Military  Deputy,  Office  of  the  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  nonconcurred  and  stated  that: 

In  this  case,  the  audit  team  found  no  such  evidence,  nor  has  any  difficulty  of  this  type 
been  reported  to  SAF/AQC  on  other  source  selections.  Regardless  of  the  quantity  of 
specific  past  performance  experiences  a  contractor  may  have,  the  evaluation  is  based  on 
contractor  past  performance  as  a  whole,  and  not  by  comparing  specific  quantities  of 
performance  experiences  for  individual  contractors. 

Audit  Response.  Management  comments  are  partially  responsive.  We  agree  that 
the  evaluation  of  source  selections  should  be  based  upon  contractor  performance 
as  a  whole.  We  also  agree  that,  in  this  case,  we  found  no  evidence  of  prejudice  in 
the  past  performance  evaluation  based  on  the  number  of  contract  experiences 
considered.  However,  in  other  source  selections,  significant  disparities  in  past 
performance  experiences  could  lead  the  SSA  to  prejudicially  rate  an  offeror  for 
past  performance.  For  example,  using  the  current  methodology,  offeror  A  could 
be  detrimentally  affected  in  a  source  selection  where  offeror  A  was  found  to  have 
two  significant  weaknesses  in  33  past  contracts  reviewed  compared  to  offeror  B, 
who  was  found  to  have  one  significant  weakness  in  13  past  contracts  reviewed. 

In  an  otherwise  even  source  selection  evaluation,  the  SSA  could  conclude  that 
offeror  B  was  the  better  offeror  because  it  had  only  one  significant  weakness 
compared  to  two  significant  weaknesses  for  offeror  A  in  past  performance.  Past 
performance  evaluation  methodology  needs  to  consider  the  impact  of  past 
performance  strengths  and  weaknesses  in  proper  proportion  to  the  population  of 
past  performance  experiences  reviewed. 

In  addition,  the  Air  Force  statement  that  no  difficulties  with  past  performance 
evaluations  have  been  reported  to  SAF/AQC  on  other  source  selections  provides 
little  assurance  that  the  difficulties  would  be  identified  in  the  absence  of  an 
independent  source  selection  review  process  (the  Air  Force  addressed  procedures 
to  improve  the  review  process  in  response  to  Recommendations  l.a.  and  l.b.). 
Thus,  we  request  that  the  Air  Force  reconsider  its  position  and  provide  additional 
comments  on  the  final  report. 
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Appendix  A.  Scope  and  Methodology 


We  reviewed  CMIS  Phase  I  and  Phase  II  source  selection  documentation  as  well 
as  CMIS  Phase  II  protest  documentation  contained  in  files  located  at  the  Air 
Force  Space  and  Missile  Systems  Center  (SMSC)  in  Los  Angeles,  California. 
Specifically,  we  reviewed  and  analyzed  SSA  changes  made  to  the  SSET  and 
SSAC  evaluation  ratings  of  the  prospective  offerors,  Boeing  Satellite  Systems  and 
Ball  Aerospace,  and  Technologies  Corporation.  Further,  we  interviewed 
available  CMIS  Phase  II  SSET  and  SSAC  representatives  located  at  the  SMSC 
and  IPO,  as  well  as  Defense  Contract  Management  Agency  representatives. 

We  performed  this  audit  from  April  2005  through  February  2006  in  accordance 
with  generally  accepted  government  auditing  standards.  Our  scope  was  limited  in 
that  we  did  not  include  tests  of  management  controls  because  the  Under  Secretary 
of  Defense  requested  that  we  specifically  review  the  NPOESS-CMIS  contract. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Government  Accountability  Office  High-Risk  Area.  The  Government 
Accountability  Office  has  identified  several  high-risk  areas  in  DoD.  This  report 
provides  coverage  of  the  contract  management  high-risk  area. 


Prior  Coverage 


During  the  last  5  years,  the  Government  Accountability  Office  (GAO)  and  the 
Defense  Contract  Management  Agency  have  issued  two  reports  and  one 
memorandum  related  to  NPOESS.  In  addition,  the  Office  of  the  Secretary  of 
Defense  has  issued  one  report  related  to  management  oversight  of  the  acquisition 
process.  Unrestricted  GAO  reports  can  be  accessed  over  the  Internet  at 
http://www.gao.gov. 


GAO 


Report  No.  GAO-06-249T,  “Polar-Orbiting  Operational  Environmental  Satellites: 
Technical  Problems,  Cost  Increases,  and  Schedule  Delays  Trigger  Need  for 
Difficult  Trade-off  Decisions,”  November  16,  2005 

Report  No.  GAO-01-1054,  “Polar-Orbiting  Environmental  Satellites  - 
Information  on  Program  Cost  and  Schedule  Changes,”  September  30,  2004 

Office  of  the  Secretary  of  Defense 

Defense  Science  Board  Task  Force  Report,  “Management  Oversight  in 
Acquisition  Organizations,”  March  2005 
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Defense  Contract  Management  Agency 

Anomaly  Memorandum,  “National  Polar-Orbiting  Operational  Environmental 
Satellite  System  -  Conical  Microwave  Imager/Sensor,”  March  2005 
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Appendix  B.  CMIS  Phase  II  Source  Selection 

Plan 


The  CMIS  Phase  II  source  selection  was  conducted  in  accordance  with  the  CMIS 
Source  Selection  Plan  (SSP),  “National  Polar-Orbiting  Operational 
Environmental  Satellite  System  -  Sensor  Payload  &  Algorithm  Development 
Down-Selection  for  the  Conical  Microwave  Imager/Sounder,”  on  February  22, 
2001  and  approved  by  the  SSA.  The  source  selection  team  used  FAR  Part  15  and 
AFFARS  Part  5315  as  guidance  throughout  the  acquisition  process. 

Organization.  The  CMIS  Phase  II  source  selection  organization  consisted  of  the 
following: 

•  SSA.  Official  designated  to  make  the  source  selection  decision  and 
document  the  decision  in  the  Source  Selection  Decision  Document. 

•  Source  Selection  Advisory  Council  (SSAC).  A  group  of  senior 
Government  personnel  who  provided  counsel  during  the  source  selection 
process  and  prepared  a  comparative  analysis  of  the  SSET’s  evaluation 
results  in  the  Proposal  Analysis  Report. 

•  SSET.  A  group  of  Government  and  non-Govemment  personnel  who 
evaluated  proposals  and  prepared  the  Proposal  Analysis  Report  to  report 
findings  to  the  SSAC  and  the  SSA.  SSET  personnel  represented  the 
various  functional  disciplines  relevant  to  the  acquisition. 

•  Non-Govemment  Personnel.  Non-Government  advisory  organizations 
that  participated  in  the  source  selection.  The  expertise  of  personnel  from 
these  organizations  -  members  of  the  Federally  Funded  Research  and 
Development  Corporation  were  required  to  support  the  evaluation  of  the 
various  functional  disciplines  associated  with  the  evaluation  factors  and 
evaluation  concepts. 

Evaluation  of  Proposals.  The  SSET  evaluated  the  offerors  based  on  the 
following  evaluation  factors. 

•  Past  Performance  (Factor  1).  Relevant  current  and  past  performance  to 
consider  the  demonstrated  record  of  performance  of  the  offeror’s  team  in 
supplying  products  and  services  that  meet  users’  needs,  including  cost  and 
schedule. 

•  Mission  Capability  (Factor  2).  Offeror’s  ability  to  satisfy  the  Statement  of 
Objectives  through  a  comprehensive  and  sound  program  approach. 

•  Proposal  Risk  (Factor  3).  Risk  associated  with  the  offeror’s  proposed 
approach  in  accomplishing  the  requirements  of  the  SSP. 
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•  Cost  (Factor  4).  Realism  of  the  compatibility  of  the  proposed  cost  with 
the  proposal  scope  and  efforts,  ground  rules  and  assumptions,  and 
schedule.  Reasonableness  of  the  offeror’s  proposal. 

Factor  1  was  of  equal  importance  to  Factor  2,  and  Factor  3  was  of  equal 
importance  to  Factor  4.  Factors  1  and  2  were  significantly  more  important  than 
Factors  3  and  4. 

In  addition,  the  SSET  evaluated  the  following  subfactors  within  Factors  1,  2, 
and  3: 

•  System  Optimization  (Subfactor  1).  Compatibility  with  the  offeror’s 
Integrated  Master  Plan,  Integrated  Master  Schedule,  and  the 
Government’s  budget  estimate  profile,  and  the  projected  performance  of 
the  offeror’s  system  (sensor  and  algorithms)  in  meeting  assigned 
Environmental  Data  Records. 

•  Sensor  Design  (Subfactor  2).  Offeror’s  ability  to  build  an  instrument  that 
will  meet  the  requirements  described  in  its  performance  specification. 

•  Science  Algorithm  and  Associated  Research  Grade  Code  Development 
(Science  Algorithm)  (Subfactor  3).  Offeror’s  compliance  with  the  Sensor 
Requirements  Document  and  derived  EDR  sensor  inputs. 

•  System  Engineering,  Integration,  and  Test  (SEIT)  (Subfactor  4). 

Soundness  of  the  offeror’s  approach  to  SEIT. 

•  Program  Execution  (Subfactor  5).  Offeror’s  ability  to  identify  and  balance 
performance,  cost,  schedule,  and  risk  mitigation  efforts  for  the  detailed 
design  and  fabrication  phase. 

The  SSET  Performance  Risk  Assessment  Group  (PRAG)  evaluated  offerors’  past 
performance  relevancy  at  the  five  subfactor  levels,  as  well  as  cost  performance  for 
the  Past  Perfonnance  Factor  (Factor  1)  evaluation.  The  SSET  Mission  Capability 
and  Proposal  Risk  Factor  teams  evaluated  and  rated  the  five  subfactors  separately 
within  the  Mission  Capability  Factor  (Factor  2),  and  assessed  a  Proposal  Risk 
Factor  (Factor  3)  rating  that  characterized  the  risk  associated  with  the  offeror’s 
proposed  approach  in  accomplishing  the  requirements  of  the  SSP.  The  SSET 
Cost  Factor  (Factor  4)  evaluators  assessed  the  offerors’  proposed  cost  based  on 
realism  and  reasonableness.  Subfactors  1  and  2  were  equal  and  most  important. 
Subfactors  3,  4,  and  5  were  of  equal  importance  and  were  individually  less 
important  than  sub  factors  1  and  2. 
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Appendix  C.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Director,  Acquisition  Resources  and  Analysis 
Under  Secretary  of  Defense  (Comp trailer)/ Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Director,  Program  Analysis  and  Evaluation 

Director,  Defense  Procurement  and  Acquisition  Policy 

Assistant  Secretary  of  the  Army  (Financial  Management  and  Comptroller) 


Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 

Commander,  Marine  Corps  Logistics  Command 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 


Combatant  Command 

Inspector  General,  U.S.  Joint  Forces  Command 


Other  Defense  Organizations 

Director,  Defense  Contract  Management  Agency 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Anned  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency  and  Financial  Management,  Committee 
on  Government  Reform 

House  Subcommittee  on  National  Security,  Emerging  Threats,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Information  Policy,  Intergovernmental  Relations, 
and  the  Census,  Committee  on  Government  Refonn 
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Department  of  the  Air  Force  Comments 


DEPARTMENT  OF  THE  AIR  FORCE 
WASHINGTON  DC 


OFFICE  OF  THE  ASSISTANT  SECRETARY  3  0  MAY  2006 


MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  INSPECTOR  GENERAL 

ATTN:  DEPtTY  INSPECTOR  GENERAL  FOR  AUDITING 

FROM:  SAF/AQ 

SUBJECT:  Draft  Audit  Repair,  Dated  20  Apr  06,  Project  No.  D2005-DOOOEE-0168 

Source  Selection  for  the  National  Polar-Orbiting  Operational  Environmental 
Satellite  System  -  Conical  Microwave  Imager'S  oundcr  (NPOESS-CMIS) 

Thank  you  for  the  time  and  effoit  your  team  invested  in  examining  the  NPOESS-CMIS 
source  selection.  I  also  appreciate  their  willingness  to  tneel  with  my  staff  to  discuss  the  draft 
report.  Our  comments  ate  as  follows: 

a.  Recommendation  1(a) -Concur.  In  response  to  previous  audits  and 
recommendations  of  the  Defense  Science  Board  Task  Force,  we  have  updated  our  contract 
clearance  procedures  to  ensure  additional  oversight  and  independent  review  of  contract  actions. 
This  oversight  will  always  be  at  least  one  level  above  the  contracting  officer  to  ensure  procedures 
in  Federal  Acquisition  Regulation  (EAR)  Part  1,  Subpan  1.6,  as  well  as  all  other  relevant 
contracting  policies  are  being  followed.  These  procedures  were  updated  via  Air  Force  Acquisition 
Circular  (AFAC)  2005-0810  on  10  Aug  05. 

b.  Recommendation  1(b)  -  Concur.  In  addition  to  the  clearance  process  changes 
addressed  in  response  to  1(a)  above,  we  recently  updated  our  legal  review  procedures  to  ensure 
legal  participation  and  review  of  key  documents  including  all  source  selection  documentation. 

This  change  to  our  legal  review  procedures  was  issued  via  AFAC  2006-0329  on  29  Mar  06. 

c.  Recommendation  2  -  Concur  As  discussed  in  the  first  recommendation, 
AFAC  2005-0810  made  several  changes  lo  the  source  selection  process.  The  changes  included  a 
requirement  for  the  source  selection  evaluation  team  (SSET)  to  provide  a  source  selection 
recommendation  to  the  source  selection  authority  (SSA).  Other  changes  included  the  retention  of 
all  relevant  records  supporting  the  SSET  recommendation  to  ensure  there  is  an  audil  trail  of  source 
selection  evaluations  after  the  information  is  presented  in  any  form  to  the  SSA. 
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d.  Recommendation  3  -  Non-concur.  The  basis  for  this  recommendation  is 
unclear  and  it  appears  to  be  speculative.  The  recommendation  seems  to  be  based  on  a  single 
statement  in  the  proposal  analysis  report  (PAR)  that  a  significant  disparity  in  past  contract 
experience  led  "to  more  areas  for  the  Govcmmcni  to  find  problems”  with  Ball  Aerospace  and 
Technologies  Corporation’s  past  performance  In  particular  the  audit  stated  that:  at  the  auditors 
could  not  determine  the  intent  oT  the  PAR  provision;  b)  SSET  personnel  could  not  recall  any 
specifics  about  such  provision:  and  c)  the  auditors  found  no  evidence  that  the  disparity  in  past 
performance  experience  was  prejudicial  to  past  performance  ratings  or  the  evaluation.  However, 
having  tdcntilied  no  prejudice  in  the  past  performance  evaluation  based  on  the  number  of 
contract  experiences  considered,  the  audit  goes  on  to  suggest  that  perhaps  significant  disparities 
in  past  contract  experience  could  lead  to  a  prejudicial  evaluation.  In  this  case,  the  audit  team 
found  no  such  evidence,  nor  has  any  difficulty  of  this  type  been  reported  to  SAF/AQC  on  other 
source  selections.  Regardless  of  the  quantity  of  specific  past  performance  experiences  a 
contractor  may  have,  the  evaluation  is  based  on  contractor  past  pertormance  as  a  whole,  and  not 
by  comparing  the  number  received  or  obtained  for  individual  contractors.  The  goal  is  to  obi  a  in 
and  evaluate  relevant  and  recent  past  performance  and  this  is  w'hal  our  guidance  describes. 

Please  refer  to  our  Past  Performance  Evaluation  Guide  dated  July  05  located  at 
https://!  34.205.252. 1 94/saf/aq/eomracting/guidesiffP. 

1  he  S AF/AQ  staff  stands  ready  to  assist  in  any  way  to  ensure  all  concerns  are  addressed 
prior  to  the  final  report  publication.  My  point  of  contact  is  Lt  Col  Julian  Thrash,  and  he  may  be 
reached  at  either  iulian.Jiiudito  pcniaiton.af.mil  or  703-588-7075 


Military  Deputy,  Office  of  the  Assistant  Secretary 
of  the  Air  Force  (Acquisition) 
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Team  Members 

The  Department  of  Defense  Office  of  the  Deputy  Inspector  General  for  Auditing, 
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